14 research outputs found
Phase Harmonic Correlations and Convolutional Neural Networks
A major issue in harmonic analysis is to capture the phase dependence of
frequency representations, which carries important signal properties. It seems
that convolutional neural networks have found a way. Over time-series and
images, convolutional networks often learn a first layer of filters which are
well localized in the frequency domain, with different phases. We show that a
rectifier then acts as a filter on the phase of the resulting coefficients. It
computes signal descriptors which are local in space, frequency and phase. The
non-linear phase filter becomes a multiplicative operator over phase harmonics
computed with a Fourier transform along the phase. We prove that it defines a
bi-Lipschitz and invertible representation. The correlations of phase harmonics
coefficients characterise coherent structures from their phase dependence
across frequencies. For wavelet filters, we show numerically that signals
having sparse wavelet coefficients can be recovered from few phase harmonic
correlations, which provide a compressive representationComment: 26 pages, 8 figure
Efficient Per-Example Gradient Computations in Convolutional Neural Networks
Deep learning frameworks leverage GPUs to perform massively-parallel
computations over batches of many training examples efficiently. However, for
certain tasks, one may be interested in performing per-example computations,
for instance using per-example gradients to evaluate a quantity of interest
unique to each example. One notable application comes from the field of
differential privacy, where per-example gradients must be norm-bounded in order
to limit the impact of each example on the aggregated batch gradient. In this
work, we discuss how per-example gradients can be efficiently computed in
convolutional neural networks (CNNs). We compare existing strategies by
performing a few steps of differentially-private training on CNNs of varying
sizes. We also introduce a new strategy for per-example gradient calculation,
which is shown to be advantageous depending on the model architecture and how
the model is trained. This is a first step in making differentially-private
training of CNNs practical
A Rainbow in Deep Network Black Boxes
We introduce rainbow networks as a probabilistic model of trained deep neural
networks. The model cascades random feature maps whose weight distributions are
learned. It assumes that dependencies between weights at different layers are
reduced to rotations which align the input activations. Neuron weights within a
layer are independent after this alignment. Their activations define kernels
which become deterministic in the infinite-width limit. This is verified
numerically for ResNets trained on the ImageNet dataset. We also show that the
learned weight distributions have low-rank covariances. Rainbow networks thus
alternate between linear dimension reductions and non-linear high-dimensional
embeddings with white random features. Gaussian rainbow networks are defined
with Gaussian weight distributions. These models are validated numerically on
image classification on the CIFAR-10 dataset, with wavelet scattering networks.
We further show that during training, SGD updates the weight covariances while
mostly preserving the Gaussian initialization.Comment: 56 pages, 10 figure
Scale Dependencies and Self-Similarity Through Wavelet Scattering Covariance
We introduce a scattering covariance matrix which provides non-Gaussian
models of time-series having stationary increments. A complex wavelet transform
computes signal variations at each scale. Dependencies across scales are
captured by the joint covariance across time and scales of complex wavelet
coefficients and their modulus. This covariance is nearly diagonalized by a
second wavelet transform, which defines the scattering covariance. We show that
this set of moments characterizes a wide range of non-Gaussian properties of
multi-scale processes. This is analyzed for a variety of processes, including
fractional Brownian motions, Poisson, multifractal random walks and Hawkes
processes. We prove that self-similar processes have a scattering covariance
matrix which is scale invariant. This property can be estimated numerically and
defines a class of wide-sense self-similar processes. We build maximum entropy
models conditioned by scattering covariance coefficients, and generate new
time-series with a microcanonical sampling algorithm. Applications are shown
for highly non-Gaussian financial and turbulence time-series
Kymatio: Scattering Transforms in Python
The wavelet scattering transform is an invariant signal representation
suitable for many signal processing and machine learning applications. We
present the Kymatio software package, an easy-to-use, high-performance Python
implementation of the scattering transform in 1D, 2D, and 3D that is compatible
with modern deep learning frameworks. All transforms may be executed on a GPU
(in addition to CPU), offering a considerable speed up over CPU
implementations. The package also has a small memory footprint, resulting
inefficient memory usage. The source code, documentation, and examples are
available undera BSD license at https://www.kymat.io
An Empirical Analysis on the Vulnerabilities of End-to-End Speech Segregation Models
International audienceEnd-to-end learning models have demonstrated a remarkable capability in performing speech segregation.Despite their wide-scope of real-world applications, little is known about the mechanisms they employ to group and consequently segregate individual speakers. Knowing that harmonicity is a critical cue for these networks to group sources, in this work, we perform a thorough investigation on ConvTasnet and DPT-Net to analyze how they perform a harmonic analysis of the input mixture. We perform ablation studies where we apply low-pass, high-pass, and band-stop filters of varying pass-bands to empirically analyze the harmonics most critical for segregation. We also investigate how these networks decide which output channel to assign to an estimated source by introducing discontinuities in synthetic mixtures. We find that end-to-end networks are highly unstable, and perform poorly when confronted with deformations which are imperceptible to humans. Replacing the encoder in these networks with a spectrogram leads to lower overall performance, but much higher stability. This work helps us to understand what information these network rely on for speech segregation, and exposes two sources of generalization-errors. It also pinpoints the encoder as the part of the network responsible for these generalization-errors, allowing for a redesign with expert knowledge or transfer learning
Maximum-entropy Scattering Models for Financial Time Series
International audienceModeling time series with complex statistical properties such as heavy-tails, long-range dependence, and temporal asymmetries remains an open problem. In particular, financial time series exhibit such properties. Existing models suffer from serious limitations and often rely on high-order moments. We introduce a wavelet-based maximum entropy model for such random processes, based on new scattering and phase-harmonic moments. We analyze the model's performance with a synthetic multifractal random process and real-world financial time series. We show that scattering moments capture heavy tails and multifractal properties without estimating high-order moments. Further, we show that additional phase-harmonic terms capture temporal asymmetries